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Executive  Summary 


The  current  effort  focused  on  providing  a  Transfer  of  Training  (ToT)  framework,  as  this 
method  provides  rigor  in  assessing  training  transfer  from  an  experimental  design 
perspective  (Lathan  et  ah,  2002).  The  first  step  was  to  develop  a  VE-based  spatial 
knowledge  acquisition  task  with  specific  training  objectives.  From  this,  both  outcome 
measures  to  illustrate  performance  effects  through  exposure  to  VE  training  and  process 
metrics  to  better  understand  these  effects  were  developed,  and  finally,  a  training  transfer 
study  was  designed.  The  goal  of  this  report  is  to  present  the  research  design  and  a  set  of 
performance  measures  as  a  toolkit  for  conducting  similar  studies.  Preliminary  results  are 
presented,  though  no  statistical  analyses  were  conducted. 

The  training  transfer  study  focused  on  a  group  of  student  navigators  at  HS-10  (San 
Diego,  California)  preparing  for  their  first  operational  rotary  wing  tactical  overland  flight, 
aimed  at  developing  and  utilizing  terrain  association  skills.  A  virtual  environment 
helicopter  (VEHEEO)  was  designed  to  allow  practice  of  these  skills.  In  order  to  assess 
the  training  transfer  effectiveness  of  the  VEHEEO,  a  set  of  outcome  and  process  metrics 
for  a  spatial  terrain  association  task  (e.g.,  ability  to  correlate  topographical  map  features 
to  out-the-window  views)  was  specified  (see  Table  2).  As  VE  training  effectiveness  has 
been  found  to  be  dependent  on  individual  difference  variables  such  as  spatial  ability 
(Waller,  1999),  several  additional  standardized  metrics  were  utilized  to  assess  process 
measures,  including  spatial  ability  (Ekstrom,  French,  &  Harman,  1976),  navigation  skill 
(Hegarty,  Richardson,  Montello,  Eovelace,  &  Subbiah,  2002),  workload  (Hart  & 
Staveland,  1988),  self-efficacy  (Scott,  2000),  and  simulator  sickness  (Kennedy,  Fane, 
Berbaum,  &  Eilienthal,  1993).  To  assess  terrain  association  skill,  a  test  was  developed  in 
which  participants  were  asked  to  correlate  out-the-window  views  with  topographical 
maps  and  vice  versa.  Eandmark  recognition  was  assessed  by  asking  participants  to  watch 
a  video  clip  of  the  flight  and  to  identify  terrain  features  they  recognized.  Global 
positioning  data  (GPS)  and  flight  communications  were  collected  during  flight. 
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Objective 


The  objective  of  this  study  was  to  assess  the  effectiveness  of  spatial  knowledge  and 
terrain  association  transfer  from  a  VE  training  environment,  and  to  determine  how  spatial 
knowledge,  developed  through  navigation  training,  may  influence  this  transfer  of 
knowledge  to  the  real  world.  In  developing  this  training  transfer  framework,  both 
outcome  measures  to  illustrate  performance  effects  through  exposure  to  VE  training  and 
process  metrics  to  better  understand  these  effects  were  used  to  evaluate  transfer  of 
training. 

Background 

Stanney,  Mourant  and  Kennedy  (1998,  p.  330)  suggest  that:  “To  justify  the  use  of  VE 
technology  for  a  given  task,  when  compared  to  alternative  approaches,  the  use  of  a  VE 
should  improve  task  performance  when  transferred  to  the  real-world  task  because  the  VE 
system  capitalizes  on  a  fundamental  and  distinctively  human  sensory,  perceptual, 
information  processing,  or  cognitive  capability.”  Eew  studies,  however,  have 
characterized  the  types  of  tasks  or  training  activities  for  which  the  unique  characteristics 
of  VEs  (i.e.,  egocentric  perspective,  stereoscopic  3D  visualization,  real-time  interactivity, 
immersion,  multi-sensory  feedback)  can  be  leveraged  to  provide  significant  gains  in 
human  performance,  knowledge,  or  experience  (Stanney  &  Zyda,  2002).  One  area  in 
which  VE  has  shown  training  enhancements,  though  not  pervasively  so,  is  that  of  spatial 
knowledge  acquisition.  Darken  and  Banker  (1998)  found  that  a  VE  could  be  used  to 
familiarize  individuals  with  unknown  environments.  In  particular,  training  gains  were 
found  for  intermediate  orienteers,  as  compared  to  advanced  or  beginner  orienteers. 

Waller  (1999)  found  that  spatial  ability,  especially  spatial  visualization  and  spatial 
orientation  ability,  affected  VE  transfer  effectiveness.  Interestingly,  users’  proficiency 
with  the  virtual  interface  (a  joystick)  was  found  to  be  the  most  influential  factor  in 
determining  transferability  of  spatial  knowledge.  Taken  together,  the  Darken  and  Waller 
studies  suggest  that  while  VE  training  can  enhance  spatial  knowledge  of  real  world 
environments,  both  individual  and  VE  system  factors  may  temper  this  transfer,  likely  due 
to  finite  cognitive  resources.  Nevertheless,  these  findings  provide  impetus  for  further 
study  of  VE  transfer-of-training.  In  particular,  it  is  essential  to  determine  if  greater 
spatial  knowledge  acquisition  can  be  obtained  if  one  controls  for  the  virtual  interface. 

The  current  study  investigated  the  effectiveness  of  VE  training  to  enhance  spatial 
knowledge  of  terrain  landmarks,  while  removing  the  need  to  directly  interact  with  the  VE 
system  by  using  an  intermediary  (i.e.,  instructor  pilot)  to  enact  navigational  commands 
communicated  by  a  navigator,  an  interaction  termed  ‘passive  navigation.’  The  study 
assessed  how  well  the  navigator  can  glean  spatial  knowledge  of  terrain  from  the  VE  and 
transfer  that  to  real-world  navigation.  The  study  further  investigated  how  spatial  ability 
may  influence  spatial  knowledge  acquisition. 

Research  Design 

While  there  are  many  methods  to  assess  training  transfer,  the  Transfer  of  Training  (ToT) 
framework  provides,  from  an  experimental  design  perspective,  the  greatest  rigor  in 
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assessing  training  transfer  (Lathan  et  al.,  2002).  This  method  estimates  simulator 
effeetiveness  as  a  eomparison  of  two  groups  of  trainees,  an  experimental  group  that 
reeeives  simulator  training  and  a  eontrol  group  that  reeeives  eonventional  training.  In  the 
eurrent  study,  the  latter  eonsisted  of  ‘best  praetiees’  training. 

In  this  study,  the  task  involved  navigating  a  land-based  route  while  flying  in  a  helieopter. 
Partieipants  were  randomly  assigned  to  one  of  two  training  groups,  (l)olassroom  training 
on  ‘best  praetiees’  for  navigation  training,  or  (2)  VE  training  which  allowed  for 
navigation  practice  of  a  virtual  model  of  the  real  flight  path  (Table  1;  note;  both  groups 
received  the  same  CSAR  ground  training  prior  to  this  experiment).  After  training,  all 
groups  navigated  the  real  world  flight  path  with  an  instructor  pilot. 

As  best  practices  training  is  focused  on  the  development  of  procedural  skills,  and  current 
VEs  provide  interactive  practice  opportunities,  it  was  hypothesized  that  groups  who 
received  VE  training  flights  would  perform  the  required  navigation  skills  more 
effectively  with  lower  perceived  workload/stress  than  those  who  received  the  best 
practices  training  regimen.  The  key  to  the  effectiveness  of  VE  spatial  navigation  training 
may  be  in  the  nature  in  which  it  couples  the  human  to  the  training  medium.  More 
specifically,  the  VE  forces  trainees  to  turn  their  heads  to  achieve  views  of  the  terrain 
being  traversed.  Chance  et  al.  (1999)  has  suggested  that  individuals  are  better  able  to 
maintain  orientation  when  they  must  turn  their  head  to  update  their  spatial  awareness. 
This  directional  cuing  ejfect  has  also  been  demonstrated  with  auditory  localization 
accuracy,  which  is  enhanced  when  an  observer  must  visually  look  in  the  direction  of  an 
auditory  target  (Jones  &  Kabanoff,  1975). 

An  additional  hypothesis  was  that  those  participants  with  better  spatial  orientation  and 
spatial  visualization  skills  would  perform  better  than  participants  with  lower  spatial 
orientation  and  spatial  visualization  skills  (co-variables),  implicating  that  VE  training 
may  be  differentially  effective,  depending  on  the  abilities  of  the  trainee.  In  addition,  it 
was  expected  that  VE  training  provided  students  with  an  opportunity  to  synthesize 
procedural  knowledge.  Thus,  a  third  hypothesis  was  that  participants  in  the  VE  training 
group  would  develop  better  terrain  association  skills,  as  the  VE  provides  hands-on 
practice  in  correlating  map  features  to  out-the-window  terrain.  It  was  also  hypothesized 
that  those  exposed  to  the  VE  trainer  would  result  in  better  navigator  process  performance. 


Table  1.  Design  of  Study 


Best  practices 

VE 

Navigate  Reai 
worid  route 

CSAR  training  pius  best  practices 

X 

X 

CSAR  training  pius  VE  training 

X 

X 

Participants 

Three  student  navigators  in  the  HS-10  training  program  participated  in  this  study.  All 
students  were  experienced  pilots,  and  had  successfully  passed  a  NATOPS  flight  (certified 
to  co-pilot  helicopter  after  16  flights)  prior  to  this  study.  In  addition,  all  students  had 
completed  CSAR  ground  school  on  overland  navigation  training.  None,  however,  had 
previously  been  responsible  for  terrain  navigation  of  a  helicopter  in  flight. 
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Apparatus 


VE  HELO 

ChrAVE  Hardware  and  Physical  Setup.  The  current  ChrAVE  system  was  developed  as 
a  practical  intermediary  step  in  establishing  the  viability  and  usability  of  embedded 
trainers.  The  ChrAVE  acts  as  a  laboratory  from  which  to  launch  research  into  the 
psychology  and  potential  of  training  certain  tasks  via  trainers/simulators.  The  ChrAVE 
primitively  mocks  the  navigator’s  seat  and  associated  controls  from  the  helicopter.  It  is 
meant  to  be  rather  generic  to  all  helicopter  communities. 

Platform. 

Seat  &  Elight  Controls  alignment.  The  current  implementation  used  a  Elight  Eink 
Inc.  seat  and  basic  helicopter  flight  controls.  These  controls  mimicked  standard  multi¬ 
axes  game  port  input  devices  to  PCs.  Two  axes  (pitch  &  roll)  were  dedicated  to  the 
cyclic,  one  (thrust)  to  the  collective,  and  one  (yaw)  to  the  rudder  pedals.  Additionally, 
there  was  a  button  on  the  collective  that  could  be  given  specific  assignments.  The  flight 
controls  were  not  used  by  the  navigating  pilot  during  this  experiment.  They  only 
provided  aesthetically  realistic  obstacles  to  the  task  of  cockpit  management  for  the 
navigating  pilot  (see  Eigures  1  and  2). 
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Figure  1.  The  ChrAVE  Platform 
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Figure  2.  ChrAVE  Instrument  Panel  (SGL  LCD  monitor) 


Headgear:  Head  Mounted  Display.  The  Virtual  Researeh  V8  Head  Mounted  Display 
(HMD)  was  selected  to  be  used  with  this  simulator,  as  it  maintains  a  high  standard  in 
performance  among  professional  HMDs,  even  though  its  active  matrix  Liquid  Crystal 
Displays  (LCD)  have  a  Video  Graphics  Array  (VGA)  pixel  resolution  of  ((640x3)x480). 
Considering  cost  versus  performance,  HMDs  of  higher  resolution  were  far  too  costly  for 
this  research.  The  V8  provides  a  CRT  quality  image.  The  V8  allows  for  interpupillary 
adjustments  as  well  as  eye  relief  adjustments.  The  V8’s  earphones  will  not  be  used  during 
this  research  therefore  they  will  be  rotated  away  from  the  ears  above  the  headband. 

Audio  will  be  provided  by  a  surround  sound  speaker  system. 

Inputs  and  outputs  for  audio,  video,  and  power  are  handled  through  an  external  control 
box.  Red  Light  Emitting  Diodes  (LED)  indicate  ‘Power  On’  and  ‘Stereo’  modes. 
Standard  15  pin  VGA  type  connectors  accept  VGA  (640  x  480  60Hz)  inputs,  readily 
available  on  today’s  graphics  engines  and  workstations. 
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Camera.  The  eamera  used  in  this  implementation  was  an  Auto  Gain  Control  (AGC)  and 
Eleetronie  Light  Control  (ELC)  Panasonie  with  three  Charged  Couple  Deviees  (CCD), 
one  eaeh  for  red,  green,  and  blue.  The  eamera  was  mounted  on  top  of  the  HMD  to  eapture 
real  images  (of  navigator’s  map  in  hand  or  flight  displays)  that  were  fused  with  virtual 
‘out  the  window’  views. 

Lens.  The  eamera  lens  used  was  a  fixed  foeal  length  (4mm)  lens.  It  had  two  adjustment 
rings,  one  for  foeus  and  the  other  for  aperture  f/stop  settings.  Changing  the  aperture  to  a 
lower  f/stop  #  allowed  more  light  to  reaeh  the  camera  sensors  but  reduced  the  depth  of 
field. 

Motion  Tracker.  The  IS-600  Mark  2  was  used  in  this  implementation.  It  is  a  hybrid 
motion  tracker  that  utilizes  inertial  and  ultrasonic  sensing  technologies  to  provide  6-DOE. 
The  Mark  2  provides  multimode  communication  redundancy  for  inertial  and  ultrasonic 
hybrid  components.  The  inertial  system  is  comprised  of  an  InertiaCube''’’^  that  is  strapped 
to  the  user’s  headgear  and  tethered  by  wire  to  a  control  unit.  It  is  nearly  immune  from 
environmental  interference.  The  ultrasonic  system  is  comprised  of  SoniDiscs™  placed 
adjacent  to  the  InertiaCube™  on  the  user’s  headgear  and  an  X-bar  installed  overhead. 

The  SoniDiscs™  chirp  an  ultrasonic  burst  when  they  sense  an  infrared  flash  from  the  X- 
bar.  The  X-bar  is  equipped  with  microphones  on  each  of  four  pods.  When  the  X-bar 
hears  the  ultrasonic  chirp  on  the  four  pods  the  location  of  the  SoniDisc™  is  calculated  by 
the  control  unit.  The  SoniDiscs™  are  susceptible  to  interference.  They  require  line  of 
sight  communication  and  normal  indoor  environmental  light  intensities  due  to  the 
infrared  portion  of  the  system. 

Chromakey  Bluescreen  Matting.  A  backdrop  made  of  standard  entertainment  industry 
chromakey  blue  cloth  panels  was  constructed  in  such  a  fashion  so  as  to  surround  the 
mock  cockpit  from  eleven  o’clock  to  four  o’clock.  Where  necessary,  chromakey  blue 
tape  was  used  to  hide  seems. 

Lighting.  Lighting  is  by  far  the  most  temperamental  component  to  implementing 
chromakey  technology.  The  chromakey  mixer  must  perceive  the  chromakey  blue 
backdrop  (called  the  matting)  without  noise  such  as  being  unevenly  lit  and  having 
shadows.  A  number  of  fluorescent  lamps  were  placed  about  the  mock  cockpit  in  such  a 
manner  so  as  to  light  the  matting  while  not  impeding  the  navigator’s  view  of  the  matting. 
An  additional  hurdle  was  ensuring  that  the  lamps  did  not  directly  shine  into  the  camera 
lens  or  the  sonic  disks.  Although  the  sonic  disks  are  alerted  to  infra  light,  the  intensity  of 
the  fluorescent  lamps  can  create  sufficient  noise  to  disrupt  proper  motion  tracking. 

This  implementation  employed  four  fixtures  that  were  four  feet  in  length  and  four 
fixtures  that  were  two  feet  in  length.  Each  fixture  has  high  output  flicker-free  ballast  that 
operate  on  120  VAC/60Hz.  Each  fixture  also  includes  a  specular  reflector,  and  two  lamp 
bam  doors. 

Signal  Converters,  Mixers,  and  Splitters.  A  number  of  signal  converters  were  used  in  the 
system.  The  chromakey  mixer  used  in  this  implementation  requires  a  CCIR-601  signal  as 
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input.  Therefore,  both  the  foreground  (FG)  signal  (an  RGB  signal  from  the  eamera)  and 
the  baekground  (BG)  signal  (a  VGA  signal  from  the  CPU)  has  to  be  eonverted. 
Furthermore,  onee  the  FG  and  BG  signals  are  mixed,  the  CCIR-610  output  signal  has  to 
be  converted  back  to  a  VGA  (640X480)  signal  for  the  HMD. 

Task 


Student  navigators  were  recruited  immediately  before  their  first  terrain  association  flight 
in  their  curriculum  (which  followed  CSAR  ground  school).  For  their  first  flight,  students 
were  asked  to  utilize  terrain  and  man  made  features  to  facilitate  passive  navigation  of  a 
helicopter.  The  flight  lasted  approximately  4  hours,  although  the  course  of  interest  for 
this  study  included  only  the  first  45  minutes  of  the  overall  route. 

Measures 


The  following  section  describes  the  measures  for  the  current  study.  Table  2  delineates 
measures  that  were  collected  and  the  time  at  which  each  was  administered..  A  brief 
description  of  each  measure  follows. 


Table  2.  Experimental  design  and  dependent  measures. 


Dependent 

measure 

Training  Session  1 
(VE  /  best  practices) 

Training  Session  2 
(VE  /  best  practices) 

Operational  Flight 

Individual 

difference 

measures 

Pre¬ 

exposure 

Post¬ 

exposure 

Pre¬ 

exposure 

Post¬ 

exposure 

Post-flight 

Spatial 

Orientation 

X 

Spatial 

Visualization 

X 

Navigation  Skill 
questionnaire 

X 

Process 

measures 

Flight 

communications 

X 

Workload 

questionnaire 

X 

X 

X 

Self-Efficacy 

X 

X 

X 

Simulator 

Sickness 

X 

(VE 

group 

only) 

X 

(VE 

group 

only) 

X 

(VE 

group 

only) 

X 

(VE 

group 

only) 

Outcome 

Measures 

Landmark 
recognition  test 

X 

Terrain 

Association  Task 

X 

X 

X 

GPS  data 

X 

Spatial  Orientation  Test:  The  ETS  Spatial  Orientation  Test  was  completed  by  each 
participant  as  a  trait  measure  of  spatial  orientation  ability. 


Spatial  Visualization  Test:  The  ETS  Spatial  Visualization  Test  was  completed  by  each 
participant  as  a  trait  measure  of  spatial  visualization  ability. 
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Workload  questionnaire:  This  questionnaire  assessed  the  eurrent  stress  state  of  the 
student  navigator  in  relation  to  performing  terrain  navigation. 

Self-Efficacy  Questionnaire:  A  self-efficacy  questionnaire  assessed  students’ 
confidence  in  performing  a  set  of  tasks  related  to  terrain  navigation  (Scott,  2000). 

Simulator  Sickness  Questionnaire  (SSQ):  The  SSQ  was  used  to  assess  any  sickness 
symptoms  experienced  during  VE  exposure.  The  SSQ  consists  of  a  checklist  of  26 
symptoms,  each  of  which  is  related  in  terms  of  degree  of  severity  (none,  slight,  moderate, 
severe).  A  weighted  scoring  procedure  is  used  to  obtain  a  global  score  reflecting  the 
overall  discomfort  level  known  as  the  Total  Severity  (TS)  score.  The  SSQ  also  provides 
scores  on  three  subscales  representing  separable  but  somewhat  correlated  dimensions  of 
simulator  sickness  (i.e..  Nausea  [N],  Oculomotor  Disturbances  [O],  and  Disorientation 
[D]). 

Navigation  Skill  Questionnaire/Santa  Barbara  Questionnaire:  This  questionnaire 
assessed  general  navigation  skill  via  self-report. 

Instructor  Questionnaire.  Instructors  were  given  a  short  questionnaire  to  assess  the 
student’s  performance  and  use  of  best  practices  in  flight. 

Course  Review  Interview.  Participants  were  provided  with  actual  GPS  track  data  from 
their  real  world  flight  and  their  intended  route  of  flight,  and  asked  to  describe  times 
during  the  course  of  the  mission  that  they  were  off  course.  Students  were  encouraged  to 
justify  each  time  their  flight  path  differed  from  the  intended  flight  path  by  more  than 
1000m. 

Terrain  Association  Task.  Participants  were  presented  with  10  ‘  out- the-window’ views, 
each  of  which  was  coupled  with  three  distinct  maps.  Students  were  asked  to  match  the 
out-the-window  view  with  one  of  the  three  maps  (a  point  on  each  map  denoted  the 
location  of  the  out-the-window  view).  Students  were  then  presented  with  10  maps,each  of 
which  was  coupled  with  three  distinct  out-the-window  views.  Again,  students  were  asked 
to  match  the  map  to  the  appropriate  out-the-window  view.  New  stimuli  (maps  and  out- 
the-window  views)  were  used  each  time  the  test  was  administered. 

VE  Video  Review.  Participants  were  presented  with  a  video  clip  illustrating  the  course 
that  they  flew,  and  asked  to  note  any  landmarks  they  recognized  from  their  flight.  (Note: 
future  forms  of  this  evaluation  may  consider  the  use  of  utilizing  the  VE  as  a  dynamic 
method  of  assessing  spatial  memory,  rather  than  passive  video). 

Additional  Dependent  Measures 

1)  Time  that  checkpoints  were  recognized  in  the  real  world 

2)  Number  of  errors  that  were  made  in  navigating 

3)  Number  of  times  that  Navigators  were  warned  that  they  were  off  course  (level  1 
warning) 
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4)  Number  of  times  that  the  Navigators  were  shown  where  they  were  (in  the  VE,  shown 
a  GPS  traek  of  where  they  were  vs.  the  planned  route;  in  the  RW,  instruetor  pilot 
provided  verbal  direetion  of  where  they  were). 

5)  Navigator  eommunieations  (online  recording  of  cockpit  conversation;  measures  of 
navigation  process) 

6)  Instructor  questionnaires 

a.  %  of  time  that  the  student  spent  heads  up  vs.  heads  down 

b.  Scaled  use  of  best  practices 

c.  Rank  order  of  the  performance  (from  GPS  tracks)  of  all  participants 

d.  Use  of  landmarks  assessment 

Method 


Three  participants  were  randomly  placed  into  either  ‘best  practices’  training  (review  of 
best  practices  e.g.  map  management,  team  performance,  scanning  skills,  etc.)  or 
VEHELO  training  (which  allowed  practice  using  a  virtual  model  of  the  real  route).  After 
training,  all  participants  navigated  a  real  world  flight  with  an  instructor  pilot.  Best 
practices  training  focused  on  summarizing  procedural  skills,  while  VEHEEO  training 
allowed  for  consolidation  of  this  knowledge  via  interactive  practice  opportunities.  It  was 
thus  hypothesized  that  individuals  who  received  VEHEEO  training  would  develop  better 
terrain  association  skills  and  perform  navigation  skills  more  effectively  with  lower 
workload  than  those  who  received  the  best  practices  training.  A  second  hypothesis  was 
that  participants  with  better  spatial  ability  would  perform  better  than  participants  with 
lower  levels  of  spatial  skills. 


Results 

Data  collection  is  ongoing,  and  no  statistical  analyses  were  completed  due  to  the  low 
number  of  participants  (N  =  3).  Individual  results  are  presented  below  for  spatial  ability, 
self-efficacy,  workload,  and  landmark  recognition.  Due  to  ongoing  modifications  of  the 
terrain  association  test,  those  data  are  not  included.  Other  data  collected  (e.g.  GPS  and 
flight  communication  data)  are  currently  being  analyzed. 
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Spatial  Ability  Performance 


Cube  Comparisons  Surface  Development 


□  VE4 
nVE3 

□  BP 


Figure  1 :  Spatial  Ability 

Figure  1  shows  that  spatial  ability  differed  across  the  participants,  with  one  participant 
from  the  VE  training  group  having  lower  spatial  ability.  The  impact  of  this  on  flight 
performance  (assessed  via  GPS  and  communications)  will  be  examined. 

Figure  2  shows  that  after  training  and  real  world  flight,  self-efficacy  increased,  regardless 
of  training  method.  Figure  3  illustrates  how  workload  decreased  post  training,  but 
increased  slightly  for  the  best  practices  participant  post-flight.  Figure  4  illustrates 
performance  on  the  landmark  recognition  test.  As  shown,  the  best  practices  participant 
recalled  the  most  terrain  features.  This  may  be  explained  by  the  nature  of  the  best 
practices  condition,  as  the  trainee  was  instructed  to  mentally  simulate  navigating  the 
route  by  calling  out  the  landmarks. 
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Figure  2:  Self-Efficacy  Ratings 
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Figure  3.  Subjective  Workload  ratings 
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Figure  4,  Landmark  Recognition  Performance 


Discussion 

Given  the  paucity  of  ToT  studies  in  operational  environments,  this  effort  focused  on  the 
development  of  a  transfer  framework.  First,  an  examination  of  VE  capabilities  and 
interviews  with  subject  matter  experts  were  used  to  derive  training  objectives.  This 
process  indicated  the  VEHELO  would  best  be  used  to  supplement  training  for  terrain 
navigation.  Next,  training  objectives  were  used  to  develop  process  and  outcome 
measures  that  would  evaluate  changes  in  performance  due  to  training.  Einally,  a  transfer 
of  training  study  was  designed  and  implemented  at  FIS- 10.  Ongoing  data  collection  and 
future  analyses  will  illustrate  the  impact  of  VE  training  on  performance  measures.  Taken 
together,  the  results  of  this  study  may  shed  light  on  the  training  transfer  effectiveness  of 
VE  training,  as  well  as  provide  insights  into  why  enhancements,  if  any,  are  realized  via 
VE  training. 


Design  Interactive,  Inc. 


Contract  #  N000I4-03-C-0I94 


15 


References 


Chance,  S.,  Gaunet,  F.,  Beall,  A.,  &  Loomis,  J.  (1999).  Locomotion  mode  affeets  the 
updating  of  objects  encountered  during  travel.  Presence,  7(2),  168-178. 

Darken,  R.  P.,  &  Banker,  W.  P.  (1998).  Navigation  in  natural  environments;  Training 
ixansiQX  study.  Proceedings  of  VRAIS  ’98,  12-19. 

Ekstrom,  R.B.,  French,  J.W.,  &  Harman,  H.H.  (1976).  Manual  for  Kit  of  Factor 
Referenced  Cognitive  Tests.  Princeton,  NJ:  Edueational  Testing  Serviee. 

Hart,  S.,  &  Staveland,  E.  (1988).  Development  of  NASA- TEX  (Task  Eoad  Index): 

Results  of  theoretical  and  empirical  research.  In  P.  Hancock  &  N.  Meshkati  (Eds.), 
Human  Mental  Workload.  Advances  in  Psyehology,  52,  1 39- 1 83.  Amsterdam: 
North-Holland. 

Hegarty,  M.  Riehardson,  A.  E.,  Montello,  D.  R.,  Eovelaee,  K  &  Subbiah,  1.  (2002). 
Development  of  a  Self-Report  Measure  of  Environmental  Spatial  Ability. 

Intelligence,  30,  425-447. 

Jones,  B.,  &  Kabanoff,  B.  (1975).  Eye  movements  in  auditory  space  perception. 
Perception  &  Psychophysics,  17(3),  241-245. 

Kennedy,  R.,  Eane,  N.,  Berbaum,  K.,  &  Eilienthal,  M.  (1993).  Simulator  sickness 

questionnaire:  an  enhanced  method  for  quantifying  simulator  siekness.  International 
Journal  of  Aviation  Psychology,  3  (3),  203-220), 

Eathan,  C.E.,  Tracey,  M.R.,  Sebrechts,  M.M.,  Clawson,  D.M.,  &  Higgins,  G.A.  (2002). 
Using  virtual  environments  as  training  simulators:  Measuring  transfer.  In  K.M. 
Stanney  (Ed.),  Handbook  of  Virtual  Environments:  Design,  Implementation,  and 
Applications  (pp.  403-414).  Mahwah,  NJ:  Eawrence  Erlbaum  Assoeiates,  Inc. 

Scott,  W.  (2000).  Training  effectiveness  of  the  VE-UNREP:  Development  of  an  UNREP 
self-efficacy  scale  (URSE).  Technical  report  submitted  to  Naval  Air  Warfare  Center, 
Training  Systems  Division.  Orlando,  EE. 


Stanney,  K.M.,  Mourant,  R.,  &  Kennedy,  R.S.  (1998).  Human  faetors  issues  in  virtual 
environments:  A  review  of  the  literature.  Presence:  Teleoperators  and  Virtual 
Environments,  7(4),  327-351. 

Stanney,  K.M.,  &  Zyda,  M.  (2002).  Virtual  environments  in  the  21st  Century.  In  K.M. 
Stanney  (Ed.),  Handbook  of  Virtual  Environments:  Design,  Implementation,  and 
Applieations  (pp.  1-14).  Mahwah,  NJ:  Eawrence  Erlbaum  Assoeiates. 


Design  Interactive,  Inc. 


Contract  #  N000I4-03-C-0I94 


16 


Waller,  D.  A.  (1999).  An  assessment  of  individual  difference  in  spatial  knowledge  of  real 
and  virtual  environments.  Unpublished  doctoral  dissertation,  University  of 
Washington. 


Design  Interactive,  Inc. 


Contract  #  N000I4-03-C-0I94 


17 


Appendix  A:  Experimental  Protocol 
Experiment  Day  1 

During  CSAR  training,  participants  were  briefed  on  the  experimental  plan,  and  asked  to 
sign  an  informed  eonsent.  Partieipants  also  eompleted  a  biographieal  data  form,  the 
Spatial  Orientation  Test,  Spatial  Visualization  Test,  Navigation  Skill  Questionnaire,  the 
Terrain  Assoeiation  Test  and  the  workload,  and  self-effieaey  questionnaires  at  this  time.. 
Partieipants  were  then  given  time  for  map  preparation  (following  standard  HS-10 
protoeol  eoneeming  time  allotted).  This  prepared  ehart  was  used  for  VE  flight  (for  those 
exposed)  as  well  as  real  world  flight.  An  instruetor  pilot  later  assessed  the  quality  of  eaeh 
map  (above  average,  average,  below  average  or  unsatisfactory),  and  results  were  given  to 
experimenters  (this  information  was  not  entered  in  the  student’s  flight  jaeket). 

Partieipants  were  then  given  a  map  preparation  questionnaire  (e.g.  to  assess  whether  the 
map  was  built  or  borrowed). 

Experiment  Day  2 

VEHELO  training  eondition 

Partieipants  who  were  randomly  seleeted  for  VEHEEO  exposure  were  given  training  on 
how  to  use  the  VE  through  a  warm-up/praetiee  period  in  the  VE  to  familiarize  themselves 
with  the  system.  Partieipants  were  then  exposed  to  the  VE  (whieh  modeled  the  area  in 
whieh  the  initial  route  of  flight  oeeurred)  through  one  45 -minute  session,  and  asked  to 
navigate  through  the  area  hitting  set  cheekpoints  throughout  the  flight.  The  same 
instruetor  pilot  flew  the  VEHEEO  for  all  students. 

During  VE  exposure,  students  were  given  a  verbal  warning  when  they  are  1000  meters 
off  eourse.  If  students  eontinued  off  eourse  after  30  seeonds,  they  were  shown  their 
eurrent  loeation  on  their  map  and  instructed  to  get  back  on  course.  If  they  eontinued  off 
eourse  for  30  seeonds,  the  session  was  paused,  and  the  navigator  was  plaeed  at  the  last 
eheekpoint  (note,  the  session  was  not  paused  for  any  partieipants  during  training). 

Simulator  siekness  measures  were  taken  pre  and  post  VE  exposure.  Dependent  measures 
taken  during  VE  flight  ineluded  position  data  (GPS),  self-reported  positioning  (reeorded 
every  2  minutes  and  at  each  checkpoint),  and  communications  data  (online  reeording  of 
eoekpit  eonversation). 


Best  Praetiees  training  condition 

Participants  were  instrueted  on  a  number  of  best  praetiees  during  map  preparation  (e.g. 
finger  traeing,  ehanging  the  map  orientation,  or  learning  the  map  outside-to-in;  see 
Appendix  J).  This  best  praetiees  session  lasted  45  minutes. 
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Experiment  Day  3 


VEHELO  training  condition 

Participants  in  the  VE  condition  had  a  second  45-minute  training  session.  Before  and 
after  VE  exposure,  participants  were  asked  to  complete  the  simulator  sickness 
questionnaire.  The  same  dependent  measures  taken  during  their  first  VE  flight  were  again 
recorded. 

Best  Practices  training  condition 

Participants  in  the  Best  Practices  condition  had  a  second  45-minute  training  session. 
Measures  completed  after  both  training  regimens 

Both  groups  completed  the  following  measures:  the  workload  measure,  the  self-efficacy 
questionnaire,  and  Terrain  Association  Test. 

Experiment  Day  4 

Real  World  Plight 

Prior  to  flight  (after  pre-brief  session),  participants  were  given  an  instruction  sheet 
describing  how  to  operate  the  data  recorder  (communication  recorder  and  GPS  recorder); 
this  sheet  described  how  data  was  to  be  collected  for  perceived  position  in  space 
(recorded  every  2  minutes  by  student  navigator).  In-flight  dependent  measures  collected 
included  position  data  (GPS),  self-reported  positioning  (recorded  every  2  minutes  and  at 
each  checkpoint),  and  communications  data  (online  recording  of  cockpit  conversation). 

At  the  completion  of  flight,  student  navigators  returned  the  GPS  and  communication 
recorders  to  experimenters.  Instructor  pilots  were  given  a  ‘best  practices’  questionnaire 
to  rate  student’s  process  performance.  Experimenters  developed  position  maps  from 
GPS. 

Post-flight  debrief 

During  the  post-flight  brief,  participants  were  asked  to  complete  a  workload  measure,  the 
self-efficacy  questionnaire.  Terrain  Association  Test,  a  VE  Video  Review  (trainees  will 
call  out  landmarks  that  they  remember  from  flight),  and  a  Course  Review  (see  above  for 
descriptions). 
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Appendix  B:  PARTICIPANT  CONSENT  FORM 


1.  Introduction.  You  are  invited  to  participate  in  a  transfer  of  training  study  evaluating 
terrain  association  skills  during  helicopter  navigation.  With  information  gathered 
from  you  and  other  participants,  we  hope  to  discover  insight  on  the  effectiveness  of 
this  training  system.  We  ask  you  to  read  and  sign  this  form  if  you  agree  to  participate 
in  the  study.  Please  ask  any  questions  you  may  have  before  signing. 

2.  Background  Information,  The  Naval  Postgraduate  School  MOVES  Institute  is 
conducting  this  study. 

3.  Procedures,  If  you  agree  to  participate  in  this  study,  the  researcher  will  explain  the 
tasks  in  detail.  There  will  be  two  training  sessions,  each  45  minutes  in  duration, 
during  which  you  will  be  expected  to  accomplish  a  number  of  tasks  related  to  terrain 
association  navigation.  In  addition,  you  will  be  asked  to  complete  a  number  of 
questionnaires. 

4.  Compensation.  No  tangible  reward  will  be  given.  A  copy  of  the  results  will  be 
available  to  you  at  the  conclusion  of  the  experiment. 

5.  Confidentiality,  The  records  of  this  study  will  be  kept  confidential.  No  information 
will  be  publicly  accessible  which  could  identify  you  as  a  participant. 

6.  Voluntary  Nature  of  the  Study,  If  you  agree  to  participate,  you  are  free  to  withdraw 
from  the  study  at  any  time  without  prejudice.  You  will  be  provided  a  copy  of  this 
form  for  your  records. 

7.  Points  of  Contact,  If  you  have  any  further  questions  or  comments  after  the 
completion  of  the  study,  you  may  contact  the  research  supervisor.  Dr.  Rudolph  P. 
Darken  (831)  656-7588  darken@n^ps.navv.mil  or  the  NPS  Flight  Surgeon,  (831)  656- 
2660. 

8.  Statement  of  Consent,  I  have  read  the  above  information.  I  have  asked  all  questions 
and  have  had  my  questions  answered.  I  agree  to  participate  in  this  study. 


Participant’s  Signature 


Date 


Researcher’s  Signature 


Date 
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Appendix  C:  MINIMAL  RISK  CONSENT  STATEMENT 


NAVAL  POSTGRADUATE  SCHOOL,  MONTEREY,  CA  93943 
MINIMAL  RISK  CONSENT  STATEMENT 

Participant:  VOLUNTARY  CONSENT  TO  BE  A  RESEARCH  PARTICIPANT  IN: 

Team  Performance  In  a  Virtual  Environment 

1.  I  have  read,  understand  and  been  provided  "Information  for  Participants"  that  provides  the 
details  of  the  below  acknowledgments. 

2.  I  understand  that  this  project  involves  research.  An  explanation  of  the  purposes  of  the 
research,  a  description  of  procedures  to  be  used,  identification  of  experimental  procedures, 
and  the  extended  duration  of  my  participation  have  been  provided  to  me. 

3.  I  understand  that  this  project  does  not  involve  more  than  minimal  risk.  I  have  been  informed 
of  any  reasonably  foreseeable  risks  or  discomforts  to  me. 

4.  I  have  been  informed  of  any  benefits  to  me  or  to  others  that  may  reasonably  be  expected  from 
the  research. 

5.  I  have  signed  a  statement  describing  the  extent  to  which  confidentiality  of  records  identifying 
me  will  be  maintained. 

6.  I  have  been  informed  of  any  compensation  and/or  medical  treatments  available  if  injury 
occurs  and  if  so,  what  they  consist  of,  or  where  further  information  may  be  obtained. 

7.  I  understand  that  my  participation  in  this  project  is  voluntary,  refusal  to  participate  will 
involve  no  penalty  or  loss  of  benefits  to  which  I  am  otherwise  entitled.  I  also  understand  that 
I  may  discontinue  participation  at  any  time  without  penalty  or  loss  of  benefits  to  which  I  am 
otherwise  entitled. 

8.  I  understand  that  the  individual  to  contact  should  I  need  answers  to  pertinent  questions  about 
the  research  is  Professor  Rudy  Darken,  Principal  Investigator,  and  about  my  rights  as  a 
research  participant  or  concerning  a  research  related  injury  is  the  Modeling  Virtual 
Environments  and  Simulation  Chairman.  A  discussion  of  the  elements  of  this  project  and  my 
consent  has  taken  place. 

Medical  Monitor:  Flight  Surgeon,  Naval  Postgraduate  School 

MOVES  Chair,  Principal  Investigator:  R.  Darken  x7588 


Signature  of  Principal  Investigator 

Date 

Signature  of  Volunteer 

Date 

Signature  of  Witness 

Date 
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Appendix  D:  PRIVACY  ACT  STATMENT 


NAVAL  POSTGRADUATE  SCHOOL,  MONTEREY,  CA  93943 

PRIVACY  ACT  STATEMENT 

1 .  Purpose:  Performance  data  will  be  collected  to  enhance  knowledge,  and  to  develop 
tests,  procedures,  and  equipment  to  improve  the  development  of  Virtual 
Environments  for  training. 

2.  Use:  Performance  data  will  be  used  for  statistical  analysis  by  the  Departments  of  the 
Navy  and  Defense,  and  other  U.S.  Government  agencies,  provided  this  use  is 
compatible  with  the  purpose  for  which  the  information  was  collected.  Use  of  the 
information  may  be  granted  to  legitimate  non-government  agencies  or  individuals  by 
the  Naval  Postgraduate  School  in  accordance  with  the  provisions  of  the  Ereedom  of 
Information  Act. 

3.  Disclosure/Confidentiality: 

a.  I  have  been  assured  that  my  privacy  will  be  safeguarded.  I  will  be  assigned  a 
control  or  code  number  which  thereafter  will  be  the  only  identifying  entry  on 
any  of  the  research  records.  The  Principal  Investigator  will  maintain  the  cross- 
reference  between  name  and  control  number.  It  will  be  decoded  only  when 
beneficial  to  me  or  if  some  circumstances,  which  are  not  apparent  at  this  time, 
would  make  it  clear  that  decoding  would  enhance  the  value  of  the  research  data. 
In  all  cases,  the  provisions  of  the  Privacy  Act  Statement  will  be  honored. 

b.  I  understand  that  a  record  of  the  information  contained  in  this  Consent  Statement 
or  derived  from  the  experiment  described  herein  will  be  retained  permanently  at 
the  Naval  Postgraduate  School  or  by  higher  authority.  I  voluntarily  agree  to  its 
disclosure  to  agencies  or  individuals  indicated  in  paragraph  2  and  I  have  been 
informed  that  failure  to  agree  to  such  disclosure  may  negate  the  purpose  for 
which  the  experiment  was  conducted. 

c.  I  also  understand  that  disclosure  of  the  requested  information,  including  my 
Social  Security  Number,  is  voluntary. 


Signature  of  Volunteer  Name,  Grade/Rank  (if  applicable)  DOB  SSN  Date 
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Appendix  E:  WORKLOAD  SURVEY 


Here  we  are  interested  in  examining  the  experienees  that  you  think  that  you  will  have 
during  the  mission.  In  the  most  general  sense,  we  are  examining  the  sense  of  “workload” 
experieneed  during  the  mission(s). 

Workload  is  a  diffieult  eoneept  to  define  preeisely.  The  faetors  that  infiuenee  your 
experienee  of  workload  may  eome  from  several  faetors.  This  survey  is  divided  into  four 
seetions  whieh  will  serve  to  assess  workload.  As  two  seetions  deal  with  assessing 
pereeptions  of  your  workload  and  two  seetions  deal  with  assessing  your  pereeption  of 
workload,  please  read  the  instruetions  for  eaeh  seetion  earefully  before  eompleting. 
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SECTION  1 :  Pre  flight  workload 

Instructions:  Place  an  X  on  each  scale  at  the  point  that  best  represents  how  much 
workload  you  think  that  you  will  experience  during  the  mission.  Marks  must  be 
placed  inside  the  box,  not  on  the  lines. _ 

1.  Mental  Demand: 

How  much  mental  and  perceptual  activity  did  the  mission  require  of  you  (e.g.,  thinking, 
deciding,  calculating,  remembering,  looking,  searehing,  etc.)? 

Low  Medium  High 

2.  Physical  Demand: 

How  much  physical  activity  did  the  mission  require  of  you  (e.g.,  pushing,  pulling, 
turning,  eontrolling,  activating,  etc.)?  This  refers  to  you  not  your  soldier. 

Low  Medium  High 

3.  Temporal  Demand: 

How  much  time  pressure  did  you  feel  due  to  the  rate  or  pace  at  whieh  the  tasks  or  task 
elements  occurred? 


Low  Medium  High 

4,  Performance: 

How  successful  do  you  think  you  were  in  accomplishing  the  goals  of  the  task?  How 
satisfied  were  you  with  your  performanee  in  aecomplishing  these  goals? 

Bad  Average  Good 

5,  Effort: 

How  hard  did  you  have  to  work  (mentally  and  physically)  to  accomplish  your  level  of 
performance? 

Low  Medium  High 

6,  Frustration: 

How  inseeure,  discouraged,  irritated,  stressed  and  annoyed  versus  seeure,  gratified, 
content,  relaxed  and  complacent  did  you  feel  during  the  task? 

Low  Medium  High 
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SECTION  2:  Post  Flight  workload 


Instructions:  Place  an  X  on  each  scale  at  the  point  that  best  represents  the  level  of 
workload  you  experienced  during  the  mission.  The  X  must  be  placed  within  the  box, 
not  on  the  line. 

1.  Mental  Demand: 

How  much  mental  and  perceptual  activity  was  required  (e.g.,  thinking,  deciding, 
calculating,  remembering,  looking,  searching,  etc.)  by  the  team  during  your  last  mission? 


Low  Medium  High 

2,  Physical  Demand: 

How  much  physical  activity  was  required  (e.g.,  pushing,  pulling,  turning,  controlling, 
activating,  etc.)  by  the  team  during  the  last  mission? 


Low  Medium  High 

3,  Temporal  Demand: 

How  much  time  pressure  did  your  team  feel  due  to  the  rate  or  pace  at  whieh  the  tasks  or 
task  elements  oecurred? 


Low  Medium  High 


4.  Performance: 

How  succtessful  do  you  think  your  team  wasi  in  accornplishing  the  goa^ 


s  of  the  task? 


Bad  Average  Good 

5.  Effort: 

How  hard  did  the  team  have  to  work  (mentally  and  physically)  to  accomplish  its  level  of 
performanee? 


Low 


Medium 


High 


6,  Frustration: 

How  insecure,  discouraged,  irritated,  stressed  and  annoyed  versus  secure,  gratified, 
content,  relaxed  and  complaeent  did  the  team  feel  during  the  task? 


Low  Medium  High 
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Appendix  F:  Self  Efficacy  HELO  Questionnaire 

Name: _ 

This  questionnaire  lists  different  activities  associated  with  performing  helicopter 
navigation  from  the  planning  phase  to  execution  phase.  With  the  assumptions  described 
below,  please  rate  how  confident  you  are  that  you  can  perform  these  tasks  for  the  actual 
mission.  We  are  not  interested  in  your  confidence  levels  for  performing  the  simulated 
version  of  this  task,  just  your  confidence  in  performing  the  real  mission.  It  is  also 
important  that  you  rate  how  confident  you  are  that  you  can  perform  these  tasks  as  of  now, 
in  this  very  moment.  Finally,  it  is  important  that  you  rate  what  you  CAN  DO  as  opposed  to 
what  you  are  willing  to  do. 
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For  the  next  questions  assume  the  following: 

Assume  the  pilot  has  been  given  speeifie  mission  objeetives  and  eonstraints  to  inelude 
aircraft  eonfiguration,  erew  load,  area  of  operation,  and  mission  support. 

First  primary  objeetive  is  to  eomplete  the  planning  phase  of  the  task.  This  involves 
aequiring  maps,  aerial  photos,  intelligenee  data,  ete.  that  will  be  used  for  planning  flight 
paths,  spider  routes,  and  assumed  aeeuraey  and  loeation  of  assumed  threats. 

Rate  your  degree  of  confidence  under  the  column  CAN  DO  NOW  by  recording  a 
number  from  0  to  100  using  the  scale  given  below. 

CAN  YOU  DO  THIS  NOW? 

0  10  20  30  40  50  60  70  80  90  100 

Certain  Moderately  Certain 

can’t  certain  can 

do  it  can  do  it  do  it 


CAN  DO  NOW? 


1 .  Conduct  a  map  study. 

2.  Conduct  a  legend  study  (e.g.  determine  elevation  scale, 

contour  interval,  vegetation  types,  cultural  features,  populous 
areas,  magnetic  variation) 

3.  Locate  and  plot  threats,  area  of  interest,  current  flight  hazards, 
SAFE  areas  and  threat  areas 

4.  Analyze  terrain  features  using  the  prominent  recognizable 
checkpoints  method 

5.  Analyze  terrain  features  using  the  prominent  limiting  features 
method 

6.  Analyze  terrain  features  using  the  prominent  guiding  features 
method 

7.  Select  navigation  points  for  primary  and  secondary  ingress  and 
egress  routes 

8.  Calculate  distance,  time  and  fuel  for  ingress  and  egress  routes 

9.  Calculate  mission  timeline 

10.  Prepare  in  flight  guides  (e.g.  kneeboard  cards,  annotated  maps) 
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For  the  next  questions  assume  the  following: 


Assume  successful  completion  of  planning  phase  tasks  and  all  associated  objectives.  The 
second  primary  objective  is  to  prepare  the  cockpit  for  the  actual  flight.  This  begins  with 
the  pre-flight  preparation,  and  concludes  with  the  aircraft  in  the  air  beginning  the 
overland  navigation  component. 

CAN  YOU  DO  THIS  NOW? 


0  10 

20 

30 

40  50  60 

70 

80 

90  100 

Certain 

Moderately 

Certain 

can’t 

certain 

can 

do  it 

can  do  it 

do  it 

CAN  DO  NOW? 


1 1 .  Conduct  navigation  to  the  initial  point 

12.  Identify  the  ingress  point  on  map 

13.  Identify  features  to  aid  identifying  initial  point 

14.  Scan  field  of  view  for  navigation  aids 

15.  Locate  navigation  aids 

16.  Positively  identify  initial  point 

17.  Estimate  arrival  time  at  initial  point 

18.  Adjust  speed  to  arrive  at  ingress  point  on  time 

19.  Adjust  course  to  overfly  initial  point 

20.  Use  visual  aids  to  identify  ingress  point 

21.  Verify  ingress  point  with  cockpit  navigation  aids 
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For  the  next  questions  assume  the  following: 


Assume  successful  completion  of  all  preceding  tasks  and  associated  objectives.  The  last 
primary  objective  is  the  actual  in-flight  navigation  component.  Because  we  make  no 
assumptions  as  to  the  length  and  duration  of  the  flight,  nor  do  we  assume  anything  about 
the  terrain  in  question,  we  assume  a  simple  repeated  procedure  for  each  pre-planned  leg 
of  the  flight.  For  each  leg,  the  navigating  pilot  will  conduct  a  number  of  sub-tasks 
involving  orientation  to  the  environment  and  self-location.  Communication  to  the  PAC 
(pilot-at-controls)  is  included.  If  disorientation  occurs  (or  even  if  it  is  believed  to  have 
occurred),  the  sub-goal  Execute-Magellan-procedure  is  entered  which  involves  re¬ 
orienting  and  getting  back  on  route. 

CAN  YOU  DO  THIS  NOW? 

0  10  20  30 

Certain 
can’t 
do  it 

CAN  DO  NOW? 


40  50  60 

Moderately 
certain 
can  do  it 


70 


80 


90 


100 

Certain 
can 
do  it 


22.  Navigate  to  next  waypoint 

23.  Direct  flying  pilot  to  predetermined  heading  by  using  the 
landmark  method 

24.  Direct  flying  pilot  to  predetermined  heading  by  using  the 
clock  position  method 

25.  Direct  flying  pilot  to  predetermined  heading  by  using  the 
turn  &  rollout  calls  method 

26.  Adjust  navigation  needle  to  new  course 

27.  Adjust  timing  using  the  late  arrival  method  with  low 
confidence  in  navigation  solution  method 

28.  Adjust  timing  using  the  late  arrival  method  with  high 
confidence  in  navigation  solution  method 

29.  Adjust  timing  using  the  early  arrival  method 

30.  Verify  PAC  proceeding  correctly 

3 1 .  Making  corrections  for  ground  speed 

32.  Check  on  track  progress 

33.  Correct  heading 

34.  Determine  aircraft  position 

35.  Scan  heading  &  track 

36.  Align  map  with  aircraft  track 

37.  Analyze  terrain  within  field  of  view 

38.  Assess  if  salient  navigation  cues  are  in  view 

39.  Match  navigation  feature  with  map  representation 

40.  Use  the  positive  match  method  (estimate  distance  and 
bearing  to  feature,  estimate  position  on  map  based  and 
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distance  and  bearing  to  feature,  update  position  on  map) 

41.  Assess  if  there  is  a  possible  correlating  feature  in  view 
method  (estimate  map  representation  of  correlating  feature, 
compare  feature  with  map,  compare  map  with  feature  to 
verify.) 

42.  Use  positive  match  with  correlating  feature  method  to  get 
back  on  track(estimate  distance  and  bearing  to  feature, 
estimate  position  on  map  based  and  distance  and  bearing  to 
feature,  update  position  on  map) 

43.  If  no  positive  match  of  correlating  feature  method,  ability  to 
use  time-distance-heading  to  maintain  track  (fly  time- 
distance-heading,  update  aircraft  position  based  on  time 
distance  heading,  update  continue  analyzing  and  comparing 
until  found  or  lost.) 

44.  Ability  to  determine  if  lost 

45.  When  terrain  features  are  located,  the  ability  to  use  the  major 
deviation  method  (determining  new  course  to  route,  treating 
current  position  as  new  waypoint,  and  executing  navigation 
to  the  waypoint) 

46.  If  have  no  cues,  the  ability  to  query  crew  for  salient  cues 

47.  If  no  cues  reported  by  crew,  ability  to  use  the  Magellan 
procedure 

48.  Ability  to  maintain  orientation 

49.  Ability  to  follow  hand  rail  method 

50.  Use  the  Visible  intermediate  navigation  point  method 

5 1 .  Use  the  Proceed  through  ambiguous  area  method 

52.  Use  the  Time  distance  heading  method 

53.  Use  the  Confess  method 

54.  Use  the  wingman  method 

55.  Use  the  RESCORT/RESCAP  method 

56.  Use  the  Orbit  method 

57.  Use  the  NOE  method 

58.  Use  the  hover  method 

59.  Use  the  land  method 

60.  Use  the  climb  method 
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The  following  lists  some  other  types  of  activities.  Again,  for  these,  please  rate  how 
confident  you  are  that  you  can  do  them  as  of  now  under  the  column  CAN  DO.  It  is 
important  that  you  rate  what  you  CAN  DO  as  opposed  to  what  you  are  willing  to  do. 


CAN  DO  NOW? 

23.  Function  competently  as  department  head  of  a  ship.  _ 

24.  Function  competently  as  the  XO  of  a  ship.  _ 

25.  Negotiate  a  take-off  maneuver  in  a  F-14  jet.  _ 

26.  Function  competently  as  the  CO  of  a  ship.  _ 

27.  Land  a  helieopter  on  a  landing  pad.  _ 
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Appendix  G:  SANTA  BARBARA  SENSE-OF-DIRECTION  SCALE 


Name:  Today's  Date: _ 

This  questionnaire  eonsists  of  several  statements  about  your  spatial  and  navigational 
abilities,  preferenees,  and  experienees.  After  eaeh  statement,  you  should  mark  one  of  the 
eireles  on  the  seale  to  indicate  your  level  of  agreement  with  the  statement.  Circle 
"strongly  agree"  if  you  strongly  agree  that  the  statement  applies  to  you,  "strongly 
disagree"  if  you  strongly  disagree,  or  some  number  in  between  if  your  agreement  is 
intermediate.  Circle  "Neutral"  if  you  neither  agree  nor  disagree. 

1 . 1  am  very  good  at  giving  directions. 


0 

0 

0 

0 

0 

0 

0 

Strongly 

Agree 

Neutral 

Strongly 

Disagree 

2. 1  have  a  poor  memory  for  where  I  left  things. 


0 

0 

0 

0 

0 

0 

0 

Strongly 

Agree 

Neutral 

Strongly 

Disagree 

3. 1  am  very  good  at  judging  distances. 


0 

0 

0 

0 

0 

0 

0 

Strongly 

Agree 

Neutral 

Strongly 

Disagree 

4.  My  "sense  of  direction"  is  very  good. 


0 

0 

0 

0 

0 

0 

0 

Strongly 

Agree 

Neutral 

Strongly 

Disagree 

5. 1  tend  to  think  of  my  environment  in  terms  of  cardinal  directions  (N,  S,  E, 
W). 


0 

0 

0 

0 

0 

0 

0 

Strongly 

Agree 

Neutral 

Strongly 

Disagree 
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6. 1  very  easily  get  lost  in  a  new  eity. 


0 

0 

0 

0 

0 

0 

0 

Strongly 

Agree 

Neutral 

Strongly 

Disagree 

7. 1  enjoy  reading  maps. 


0 

0 

0 

0 

0 

0 

0 

Strongly 

Agree 

Neutral 

Strongly 

Disagree 

8. 1  have  trouble  understanding  direetions. 


0 

0 

0 

0 

0 

0 

0 

Strongly 

Agree 

Neutral 

Strongly 

Disagree 

9. 1  am  very  good  at  reading  maps. 


0 

0 

0 

0 

0 

0 

0 

Strongly 

Agree 

Neutral 

Strongly 

Disagree 

10. 1  don't  remember  routes  very  well  while  riding  as  a  passenger  in  a  ear. 


0 

0 

0 

0 

0 

0 

0 

Strongly 

Agree 

Neutral 

Strongly 

Disagree 

1 1. 1  don't  enjoy  giving  direetions. 


0 

0 

0 

0 

0 

0 

0 

Strongly 

Agree 

Neutral 

Strongly 

Disagree 

12.  It's  not  important  to  me  to  know  where  I  am. 


0 

0 

0 

0 

0 

0 

0 

Strongly 

Agree 

Neutral 

Strongly 

Disagree 
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13.1  usually  let  someone  else  do  the  navigational  planning  for  long  trips. 


0 

0 

0 

0 

0 

0 

0 

Strongly 

Agree 

Neutral 

Strongly 

Disagree 

14. 1  can  usually  remember  a  new  route  after  1  have  traveled  it  only  once. 


0 

0 

0 

0 

0 

0 

0 

Strongly 

Agree 

Neutral 

Strongly 

Disagree 

15.1  don't  have  a  very  good  "mental  map"  of  my  environment. 


0 

0 

0 

0 

0 

0 

0 

Strongly 

Agree 

Neutral 

Strongly 

Disagree 
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Figure  Caption 

Figure  1.  Example  of  a  eity  question  from  Study  7. 


Pretend  you  are  at  Denver,  CO,  Facing  Miami,  FL. 
Draw  an  arrow  pointing  toward  Dallas  TX. 
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Appendix  H:  Terrain  Association  Evaluation 


Terrain  association  is  the  skill  of  correlating  map  features  to  out  the  window  views  and 
vice  versa.  For  this  evaluation,  we  will  ask  you  to  match  map  views  to  out-the-window 
views. 

For  the  first  half  of  the  evaluation,  you  will  be  presented  with  a  map  and  asked  to  identify 
which  out-the-window  view  corresponds  to  the  map.  Indicate  which  of  the  3  views  is  the 
correct  answer  on  the  answer  sheet.  The  second  half  of  the  test  will  present  you  with  an 
out-the-window  view,  and  you  will  be  asked  to  identify  which  map  corresponds  with  the 
out-the-window  view.  Indicate  which  of  the  3  maps  is  the  correct  answer  on  your  answer 
sheet.  There  are  20  items,  so  please  work  as  quickly  and  accurately  as  you  can,  but  do 
not  sacrifice  accuracy  for  time. 

Please  note  that  when  you  select  an  answer,  the  program  will  automatically  direct  you  to 
the  next  item. 

Do  you  have  any  questions?  Please  tell  the  evaluator  that  you  are  ready  to  begin. 
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Subject  Name 


Appendix  I:  SSQ 

Directions:  Rate  your  experience  of  the  following  (i.e.  right  now  I  feel. . .) 


I .  General  discomfort 

None 

_ Slight _ 

Moderate 

_ Severe 

2.  Fatigue 

None 

_ Slight _ 

Moderate 

_ Severe 

3.  Headache 

None 

_  Shght_ 

Moderate 

_ Severe 

4.  Eyestrain 

None 

_ Slight _ 

Moderate 

_ Severe 

5.  Difficulty  focusing 

None 

_  Shght_ 

Moderate 

_ Severe 

6.  Increased  salivation 

None 

_ Slight _ 

Moderate 

_ Severe 

7.  Sweating 

None 

_ Slight _ 

Moderate 

_ Severe 

8.  Nausea 

None 

_ Slight _ 

Moderate 

_ Severe 

9.  Difficulty  Concentrating 

None 

_ Slight _ 

Moderate 

_ Severe 

10.  Fullness  of  Head 

None 

_  Shght_ 

Moderate 

_ Severe 

11.  Blurred  Vision 

None 

_ Slight _ 

_ Moderate _ 

_ Severe 

12.  Dizzy  (eyes  open) 

None 

_ Slight _ 

Moderate 

_ Severe 

13.  Dizzy  (eyes  closed) 

None 

_ Slight _ 

Moderate 

_ Severe 

14.  Vertigo 

None 

_  Shght_ 

Moderate 

_ Severe 

15.  Stomach  awareness 

None 

_ Slight _ 

Moderate 

_ Severe 

16.  Burping 

None 

_ Slight _ 

Moderate 

_ Severe 
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Appendix  J:  Best  Practices  (MaP  TeNS) 

MaP  TeNS 


10  Best  Practices:  Key  Points  to  Remember 

MaP  TeNS 

>  Map  Management:  Physically  manipulate  your  map  to  decrease 
workload 

>  Prioritization:  Aviate,  navigate,  communicate 

>  Team  Performance:  Use  your  team  to  be  a  another  set  of  eyes 

>  Navigation  backup  /Map  Use:  Use  control  points  and  DR  from  planning 
to  backup  your  navigation 

>  Scanning:  Don’t  get  tunnel  vision 
List  of  10  Best  Practices: 

Map  management 

1  ■  Orient  the  map  in  the  direction  of  travel 

•  Hold  map  in  position  that  facilitates  rapid  transition  from  inside  to  outside  scan 
(not  down  on  the  lap) 

•  Refold  the  map  as  you  go  along  to  follow  the  route 
Prioritization 

2,  Prioritize  tasks  (aviate,  navigate,  communicate) 

•  Focus  on  avoiding  obstacles,  fuel,  altitude,  etc.  first  (terrain  collision  avoidance) 

•  Look  out  the  window  and  on  map  for  terrain  features 

•  Communicate  with  ground  control 

Team  Performance 

3.  Provide  clear,  timely  direction  to  PAC 

•  Use  terrain  or  clock  references  to  provide  direction  instead  of  course  headings 
Best  -  fly  to  saddle  to  left  of  peak  at  your  2  o’clock 
Good  -  turn  to  2  o’clock 
Good  -  come  right,  . . .  roll  out 
Weak  -  turn  to  095 
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4,  Keep  crew  informed  of  navigation  picture:  location,  expected  hazards  &  navigation 
features,  confidence  in  navigation  solution 

•  Provide  situation  updates  (e.g.  heading,  legs,  timing,  hazards) 

•  Use  voice  inflection  (and  direct  comms)  to  let  others  know  confidence  in 
navigation  solution 

5.  Prompt  crew  for  features/hazards  that  should  be  in  their  view 

•  Ask  crew  for  backup  with  terrain  features  (e.g.  use  crewmembers  as  extra  eyes) 

•  Listen  and  acknowledge  if  crew  notes  a  terrain  feature 


Navigation  Backup  best  practices 

6.  Use  intermediate  checkpoints  and  checking  features  to  verify  navigation  and  judge 
along-track  progress 

•  In  map  planning,  identify  salient  features  to  use  as  intermediate  points 

•  In  flight,  use  intermediate  points  as  a  way  to  continually  assess  current  location 

•  In  map  planning,  identify  checking  features  to  use  as  navigation  backup 

•  In  flight,  use  checking  features  to  assess  if  you  are  off  course 

7.  Use  elevation  as  identifying  feature  when  possible 

•  In  map  planning,  use  elevation  to  help  identify  key  features 

•  In  flight,  use  elevation  to  assess  relative  orientation 

8.  Back  up  terrain  association  with  dead  reckoning  (time,  distance,  heading) 

•  Verbalize  when  approaching  checkpoint,  call  out  time  and  distance  to  turn, 
direction  of  turns  and  heading  as  backup  to  terrain  association 

Scanning  best  practices 

9.  Divide  scan  appropriately:  proportion  of  time  scanning  OTW  and  map 

•  When  looking  within  the  cockpit,  perform  an  instrument  scan,  evaluate  time, 
distance,  and  heading  and  check  map  for  mental  list  of  things  you  should  be 
seeing 

o  Determine  upcoming  terrain  features  direct  attention  to  what  you  are  most 
uncertain  of 

10  Scan  complete  FOV 

•  Keep  your  head  moving  to  avoid  tunnel  vision  (e.g.  don’t  fixate  on  instruments  or 
terrain  features) 


Design  Interactive,  Inc. 


Contract  #  N000I4-03-C-0I94 


39 


Appendix  K:  Mental  Simulation  Review  of  Course 


o  Map  review: 

■  Have  them  tell  us  eheekpoints,  intermediate  eheekpoints,  limiting 
features,  hazards  in  review 

■  Time  to  eomplete:  15  minutes 
o  Mental  simulation: 

■  Have  students  mentally  follow  the  route  and  eall  out  landmarks, 
hazards,  intermediate  eheekpoints  ete.  as  they  would  expeet  to  see 
them 

■  Provide  baekup  to  terrain  with  timing,  distanee  and  heading 
information 

■  Move  the  map  around  to  mateh  what  they’re  looking  for  (direetion 
of  travel) 

■  Refold  map  to  follow  the  route 

■  Time  to  eomplete  twiee:  30  minutes 

Note:  they  are  also  praetieing  the  best  praetiees  of  keeping  the  erew  informed  and 
prompting  the  erew  for  visual  eues  by  stating  the  landmarks/hazards  aloud 
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Appendix  L:  Instructor  Questionnaire  (Post  Flight) 

Instructor  Name _ 

Student  Navigator  on  your  flight _ 

Date  /  Time  of  flight _ 


Post  Flight  Questions  (Instruetor  Pilot) 

Please  Mark  an  “X”  along  the  seale  below  each  question.  Note  that  we  are  only 
interested  in  the  outbound  route  (e.g.  the  first  45  minutes  of  the  flight).  We  are 
speeifieally  interested  in  the  student’s  ability  to  perform  Terrain  Assoeiation  and  their  use 
of  best  praetiees. 

1 .  How  often  did  the  student  navigator  use  terrain  assoeiation  to  navigate  the  route? 

0%  25%  50%  75%  100% 

(of  the  route)  (of  the  route) 


2.  How  much  time  did  the  student  navigator  spend  in  a  ‘heads-up’  position  versus  a 
‘heads-down’  position? 

0%  25%  50%  75%  100% 

(of  the  time)  (of  the  time) 


3.  How  often  did  the  student  navigator  referenee  absolute  heading  (from  instrument 
panel)  along  the  route? 

0%  25%  50%  75%  100% 

(of  the  time)  (of  the  time) 


4.  How  often  did  the  student  navigator  prompt/query  for  visual  features  from  other 
erew  members? 

0%  25%  50%  75%  100% 

(of  the  time)  (of  the  time) 
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5.  How  often  was  the  student  navigator  ‘on  route’? 

0%  25%  50%  75% 

(of  the  time) 


6.  How  often  was  the  student  navigator  ‘on  time’? 

0%  25%  50%  75% 

(of  the  time) 


100% 

(of  the  time) 


100% 

(of  the  time) 


7.  How  would  you  rate  this  student’s  overall  navigation  performance? 

1  (best  ever)  3  (average)  5  (very  poor) 


8.  How  would  you  rate  this  student’s  map? 

1  (best  ever)  3  (average) 


5  (very  poor) 


9.  How  often  did  the  student  orient  their  map  in  the  direction  of  travel? 

0%  25%  50%  75%  100% 

(of  the  time)  (of  the  time) 


10.  Did  the  student  provide  clear  and  timely  direction  to  the  pilot? 

0%  25%  50%  75%  100% 

(of  the  time)  (of  the  time) 


1 1 .  How  often  did  the  student  keep  the  crew  informed  about  location,  expected 
hazards  and  navigation  features? 

0%  25%  50%  75%  100% 

(of  the  time)  (of  the  time) 
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12.  How  often  did  the  student  prompt  the  erew  for  location,  hazards  and  navigation 
features? 


0%  25%  50%  75%  100% 

(of  the  time)  (of  the  time) 


13.  How  often  did  the  student  make  reference  to  intermediate  checkpoints  when 
appropriate? 

0%  25%  50%  75%  100% 

(of  the  time)  (of  the  time) 


14.  How  often  did  the  student  use  checking  features,  where  needed? 

0%  25%  50%  75%  100% 

(of  the  time)  (of  the  time) 


15.  How  often  did  the  student  use  elevation  as  an  identifying  feature  when 
appropriate? 

0%  25%  50%  75%  100% 

(of  the  time)  (of  the  time) 


16.  How  often  did  the  student  scan  his  complete  FOV? 

0%  25%  50%  75%  100% 

(of  the  time)  (of  the  time) 
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Appendix  M:  Course  Review 

Participants  will  be  provided  with  aetual  GPS  traek  data  from  the  real  world  flight  and 
the  planned  route  and  asked  to  identify  landmarks  for  each  leg  and  to  describe  times 
during  the  eourse  of  the  mission  that  they  were  off  eourse. 
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Appendix  N:  Number  of  Terrain  Association  References  in  Course  Review 


Student: 

Experimenter: 


Leg  I: 

Leg  2: 

Leg  3: 

Leg  4: 

Leg  5: 

Leg  6: 

Leg  7: 

Leg  8: 

Leg  9: 
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